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Abstract. This paper aims to describe a computer-assisted English learning system 
using chatbots and dialogue systems, which allow free conversation outside the topic 
without limiting the learner’s flow of conversation. The evaluation was conducted 
by 20 experimenters. The performance of the system based on a free conversation 
by topic was measured by the success rate of the dialogue turn. The average success 
rate of the dialogue turn was 80.86% and the success rate of dialogue turns for 
each topic was as follows: (1) ‘purchasing New York city tour tickets’: 71.86%; (2) 
‘ordering food’: 71.06%; (3) ‘talking about health habits’: 85.41%; and (4) ‘thinking 
about future currency’: 95.09%. Additionally, the precision and recall of English 
grammar error correction was 66.7% and 31.9% respectively. 

Keywords: dialogue system, computer-assisted language learning, free conversation, 
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1. Introduction 

A dialogue system is a system that allows conversation between a user and a 
system in a natural voice. The dialogue system, which provides an educational 
environment similar to that of receiving English conversation education from a 
native speaker by combining the English education field with the dialogue system, 
is said to be a computer-assisted English learning system. The Korea Electronics 
and Telecommunications Research Institute has developed GenieTutor, a computer 
assisted English learning system from 2010 to 2015 (Choi, Kwon, Kim, & Lee, 
2016). While GenieTutor asks questions based on a given topic, talks to the learner 
according to the scenario of the topic, and provides feedback on grammar errors, 
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it has the disadvantage of limiting the learner's flow of conversations and does 
not allow free conversation outside the topic. To overcome these drawbacks, 
the computer-assisted English learning system described in this paper has been 
developed: GenieTutorPlus (Kwon et al., 2015). It allows free conversation 
outside the topic without limiting the learner’s flow of dialogue and feedback on 
grammatical errors. 


2. Free talk by topic 

GenieTutorPlus, as opposed to GenieTutor, aims to help learners continue their 
English learning by returning to the topic without interrupting their learning, even 
if they are talking with the dialogue system outside the topic. Therefore, the newly 
developed English learning system is a system that allows free conversation by 
topic. GenieTutor and GenieTutorPlus are summarised in Table 1. 


Table 1. Comparison between GenieTutor and GenieTutorPlus 


GenieTutor 

GenieTutorPlus 

• System-initiative 

• No free conversation out¬ 
side the topic 

• Mixed-initiative between learner and system 

• Free conversation outside the topic 

• Chatbot dialogue 


Figure 1, below, is an example of GenieTutorPlus for the topic of ‘ordering food’. 
Figure 1. Example of GenieTutorPlus for the topic of ‘ordering food’ 


1. [System] Hi. What would you like? 

2. [Learner] Do you have menu? 

3. [System] Here you are. What would you like to have? 

4. [Learner] I would like cheese omelette. 

5. [Tutor] I would like to have a cheese omelette . 

6. [System] Okay, there are side dishes you can order. 

Would you like something with your omelette? 

7. [Learner] I want to take the coffee. 

8. [System] Is this to go or will you be dining in? 

9. [Learner] For here. How much is it? 

10. [System] That will be 12 dollars and 55 cents, please. 

11. [Learner] I have no money. 

12. [System] What a pity! 12 dollars and 55 cents, please. 

13. [Learner] Here you are. 

14. [System] Thanks. Here's your order. Have a nice day. 

15. [Learner] Goodbye 
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For instance, [Tutor] refers to the result of recommending the correct sentence by 
operating the grammar error correction function of the [System]. The second line 
of Figure 1 shows a [Learner]’s utterance that is related to the topic but is not in the 
learning corpus. Line 11 shows a learner’s utterance that has nothing to do with the 
topic. While GenieTutor has an ‘awkward’ or ‘wrong’ system answer to the learner 
utterance in these two situations, GenieTutorPlus responds correctly as shown in 
Figure 1. 


3. Processing of free talk by topic 

3.1. How does GenieTutorPlus recognise 
that user utterance is off the topic? 

Whether or not the learner's utterance is out of the topic is determined by the 
topic recognition. Topic recognition is done by comparing a dialogue intention 
generated from a morphological analysis and a named entity recognition of user 
utterance with a dialogue intention learned from dialogue maps of topic (Choi et 
al., 2016). 

If a dialogue intention of the user utterance is not included in dialogue intentions 
by topic, the user utterance is considered to be outside the topic. A dialogue 
intention is obtained by machine learning and consists of dialogue act and slot. 
When the user utterance is out of topic, the chatbot responds to the user utterance. 

3.2. What is the overall flow 
of GenieTutorPlus? 

Figure 2 illustrates GenieTutorPlus’s flow, which starts from the learner’s utterance 
to determine whether it is topic-oriented or chatbot-oriented, and/or whether it has 
grammar errors. 

When a learner makes an utterance, GenieTutorPlus first confirms whether it 
is topic-oriented. If the utterance is not a topic conversation, GenieTutorPlus 
responds with the chatbot and then returns to the topic conversation. Otherwise, 
if the learner’s utterance is topic-oriented, GenieTutorPlus in turn processes 
suitability of content, grammar error detection, and error severity. If a grammatical 
error is detected, GenieTutorPlus provides feedback on the grammatical error and 
responds to the contextual sentence within the topic. 
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Figure 2. Overall flow of GenieTutorPlus 



3.3. How does GenieTutorPlus provide 
feedback on grammar errors? 

The language learning module of GenieTutorPlus automatically detects and 
corrects grammatical errors that may be included in the learner's utterances. 
Therefore, GenieTutorPlus aims to improve the learner's ability to speak a foreign 
language by feeding back the corrected result to the learner. 

• Detection of grammar errors: grammatical errors are detected in sentences 
which are spoken by the learner, and are limited to grammatical errors 
defined in the system, and multiple grammatical errors can be detected in 
one sentence. 

• Correction of grammar errors: grammar error correction information 
is generated by using the part of the error vocabulary and surrounding 
context information. 

• Feedback of grammar errors: indicates learner's errors, gives better 
expression, and uses explicit and implicit feedback effectively in context. 
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4. Evaluation 

The English language education institute is building the free conversation by topic 
for discussion and learning. Among these topics, we chose four topics to evaluate 
GenieTutorPlus. The selected topics and evaluation methods were as follows: 

• Topics for evaluation: (1) purchasing tickets for New York city tour, (2) 
ordering food, (3) talking about health habits, and (4) thinking about 
future currency. 

• There were 20 participants. Participants’ English level was beginners (ten) 
and intermediate (ten). Participants were aged in the 10’s (one), 20’s (15), 
and 30’s (4). There were 11 males and nine females. 

• Experimental method: the participants randomly chose a mission by topic. 
They communicated freely with GenieTutorPlus to accomplish their 
mission. Participants performed ten missions on four topics, each taking 
an average of 50 minutes per person. 

• Success rate of dialogue turn (%)=(number of correct system’s responses 
to learner’s utterance)/(total number of learners’ utterances) x 100. 

• Success rate of chatbot (%) = (number of correct system’s responses 
to learner’s out-of-topic utterance)/(number of learners’ out-of-topic 
utterances) x 100. 

• Precision of grammar error correction = (number of words correctly 
corrected by the system)/(total number of words the system attempted to 
correct for grammatical errors) x 100. 

• Recall of grammar error correction = (number of words that the system 
corrects correctly)/(actual grammatical error of the evaluation set total 
words) x 100. 

5. Results and discussion 

As illustrated in Table 2, the success rate of dialogue turn on the four topics is 
80.86%. The chatbot dialogue rate equals to 10.44% and the success rate of chatbot 
dialogue turn is 31.49%. Even though chatbot dialogue rate is occasional in the free 


83 




Sung-Kwon Choi, Oh-Woog Kwon, and Young-Kil Kim 


dialogue (10.44%), the success rate of chatbot dialogue turnout is rather low. This 
is mostly due to the fact that generating chatbot responses which are still suitable 
for the topic of dialogue is difficult to achieve. 


Table 2. Evaluation of GenieTutorPlus 


Topic 

Experimental 

group 

Success rate of 
Dialogue turn 

Rate of 

Chatbot 

dialogues 

Success rate 
of Chatbot 

Buying New 
York City 

Tour Tickets 

beginner 

73.75% 

7.50% 

16.67% 

intermediate 

class 

70.11% 

9.20% 

75.00% 

Total 

71.86% 

8.38% 

50.00% 

Ordering Food 

beginner 

73.66% 

6.99% 

15.38% 

intermediate 

class 

68.66% 

11.94% 

33.33% 

Total 

71.06% 

9.56% 

27.03% 

Health Habits 
Dialogue 

beginner 

80.53% 

12.39% 

42.86% 

intermediate 

class 

90.00% 

14.17% 

23.53% 

Total 

85.41% 

13.30% 

32.26% 

Thinking About 
Future Currency 

beginner 

96.45% 

14.18% 

15.00% 

intermediate 

class 

93.75% 

6.94% 

20.00% 

Total 

95.09% 

10.53% 

16.67% 


Total 

80.86% 

10.44% 

31.49% 


Moreover, 500 sentences were extracted from log data from the above experiment. 
Precision and recall of the English grammar error correction was calculated (see 

Table 3). 


Table 3. English grammar error correction 



Precision 

Recall 

English grammar error correction 

66.7% 

31.9% 


6. Conclusions 


In this paper, we describe an English learning system using chatbots and dialogue 
systems that help learners continue a dialogue in English even if they are outside 
of a given topic, and then return to the topic again. The system determines whether 
the learner's utterance is out of topic by comparing the topic itself with the dialogue 
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intention. The performance of the English learning system using chat hot and 
dialogue system was measured by the success rate of the dialogue turn and the 
success rate of the chatbot. The success rate of total dialogue turns reached 80.86% 
and the success rate of the chatbot was 31.49%. Precision and recall of grammar 
error correction equaled 66.7% and 31.9% respectively. 

In the future, we will investigate how the English learning system handles multiple 
turns when dialogue goes beyond the predefined discussion topic (Dingli & Scerri, 
2013). 

7. Acknowledgements 

This work was supported by the Institute for Information & communications 
Technology Promotion (IITP) grant funded by the Korea government (MSIT) 
(R0126-15-1117, Core technology development of the spontaneous speech 
dialogue processing for the language learning). 


References 

Choi, S. K., Kwon, O. W., Kim, Y. K., & Lee, Y. K. (2016). Using a dialogue system based on 
dialogue maps for computer assisted second language learning. In S. Papadima-Sophocleous, 
L. Bradley & S. Thouesny (Eds), CALL communities and culture - short papers from 
EUROCALL 2016 (pp. 106-112). Research-publishing.net. https://doi.org/10.14705/ 
rpnet.2016.eurocall2016.546 

Dingli, A., & Scerri, D. (2013). Building a hybrid: chatterbot-dialog system. Proceedings of 
16th International Conference of Text, Speech, and Dialogue (pp. 145-152). https://doi. 
org/10.1007/978-3-642-40585-3_19 

Kwon, O. W„ Choi, S. K, Roh, Y. H„ Kim, Y. K, Park, J. G., & Lee, Y. K. (2015). Trends of 
spontaneous speech dialogue processing. Electronics and Telecommunications Trends, 26- 
35, in Korean. 


85 





search-publishing.net 


Published by Research-publishing.net, not-for-profit association 
Contact: info@research-publishing.net 

© 2017 by Editors (collective work) 

© 2017 by Authors (individual work) 

CALL in a climate of change: adapting to turbulent global conditions - short papers from EUROCALL 2017 
Edited by Kate Borthwick, Linda Bradley, and Sylvie Thouesny 

Rights: This volume is published under the Attribution-NonCommercial-NoDerivatives International (CC BY-NC-ND) 
licence; individual articles may have a different licence. Under the CC BY-NC-ND licence, the volume is freely available 
online (https://doi.org/10.14705/rpnet.2017.eurocall2017.9782490057047) for anybody to read, download, copy, and 
redistribute provided that the author(s), editorial team, and publisher are properly cited. Commercial use and derivative 
works are, however, not permitted. 

Disclaimer: Research-publishing.net does not take any responsibility for the content of the pages written by the authors 
of this book. The authors have recognised that the work described was not published before, or that it was not under 
consideration for publication elsewhere. While the information in this book are believed to be true and accurate on the date of 
its going to press, neither the editorial team, nor the publisher can accept any legal responsibility for any errors or omissions 
that may be made. The publisher makes no warranty, expressed or implied, with respect to the material contained herein. 
While Research-publishing.net is committed to publishing works of integrity, the words are the authors’ alone. 

Trademark notice: product or corporate names may be trademarks or registered trademarks, and are used only for 
identification and explanation without intent to infringe. 

Copyrighted material: every effort has been made by the editorial team to trace copyright holders and to obtain their 
permission for the use of copyrighted material in this book. In the event of errors or omissions, please notify the publisher of 
any corrections that will need to be incorporated in future editions of this book. 

Typeset by Research-publishing.net 

Cover design based on © Josef Brett’s, Multimedia Developer, Digital Learning, http://www.eurocall2017.uk/, reproduced 
with kind permissions from the copyright holder. 

Cover layout by © Raphael Savina (raphael@savina.net) 

Photo “frog” on cover by © Raphael Savina (raphael@savina.net) 

Fonts used are licensed under a SIL Open Font License 

ISBN 13: 978-2-490057-04-7 (Ebook, PDF, colour) 

ISBN 13: 978-2-490057-05-4 (Ebook, EPUB, colour) 

ISBN13: 978-2-490057-03-0 (Paperback - Print on demand, black and white) 

Print on demand technology is a high-quality, innovative and ecological printing method; with which the book is never ‘out 
of stock’ or ‘out of print’. 

British Library Cataloguing-in-Publication Data. 

A cataloguing record for this book is available from the British Library. 

Legal deposit: Bibliotheque Nationale de France - Depot legal: decembre 2017. 



